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(54) Title: METHODS, APPARATUS AND PROGRAMS FOR GENERATING AND UTILIZING CONTENT SIGNATURES 

(57) Abstract: The present invention relates to deriving and utilizing content signature (42). A content signature is a representation 
of a content (40) item, which is derived from the content item itself. According to the invention, a method generating a content 

Q signature for a signal is provided. The method includes the steps of : i) dividing the signal into at least one set; ii) transforming (22) 
the set into a frequency-based domain; iii) determining features of the transformed set; and iv) grouping the features so as to form a 
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resulting content signature may be utilized to identify, track, manage, handle, protect 
the content, link to additional information and/or associated behavior, and etc. Content 
signatures are also known as "robust hashes" and "fingerprints," and are used 
interchangeably throughout this disclosure. 

[0006] Content signatures can be stored and used for identification of the content 
item. A content item is identified when a derived signature matches a predetermined 
content signature. A signature may be stored locally, or may be remotely stored. A 
content signature may even be utilized to index (or otherwise be linked to data in) a 

. related database. In this manner, a content signature is utilized to access additional 
data, such as a content ID, licensing or registration information, other metadata, a 
desired action or behavior, and validating data. Other advantages of a content signature 
may include identifying attributes associated with the content item, linking to other 
data, enabling actions or specifying behavior (copy, transfer, share, view, etc.), 

. protecting the data, etc. 

[0007] A content signature also may be stored or otherwise attached with the content 
item itself, such as in a header (or footer) or frame headers of the content item. 
Evidence of content tampering can be identified with an attached signature. Such 
identification is made through re-deriving a content signature using the same technique 
as was used to derive the content signature stored in the header. The newly derived 
signature is compared with the stored signature. If the two signatures fail to match (or 
otherwise coincide), the content item can be deemed altered or otherwise tampered 
with. This functionality provides an enhanced security and verification tool. 

[0008] A content signature may be used in connection with digital watermarking. 
Digital watermarking is a process for modifying physical or electronic media (e.g., 
data) to embed a machine-readable code into the media. The media may be modified 
such that the embedded code is imperceptible or nearly imperceptible to the user, yet 
may be detected through an automated detection process. Most commonly, digital 
watermarking is applied to media signals such as images, audio signals, and video 
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etc). A watermark can include a calibration and/or synchronization signal to realign 
the content to a base state. Or a technique can be used to determine a calibration and/or 
synchronization based upon the watermark data during the watermark detection 
process. This calibration signal (or technique) can be used to scale the content so it 
matches the scale of the content when the content signature was registered in a database 
or first determined, thus reducing errors in content cignature extraction. 

[0013] These and other features and advantages will become apparent with reference 
to the following detailed description and accompanying drawings. 

Brief Description of the Drawings 

[0014] Fig. 1 is a flow diagram of a content signature generating method. 

[0015] Fig. 2 is a flow diagram of a content signature decoding method. 

[0016] Fig. 3 is a diagram illustrating generation of a plurality of signatures to form a 
list of signatures. 

[0017] Fig. 4 is a flow diagram illustrating a method to resolve a content ID of an 
unknown content item. 

[0018] Fig. 5 illustrates an example of a trellis diagram. 

[0019] Fig. 6 is a flow diagram illustrating a method of applying Trellis Coded 
Quantization to generate a signature. 
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[0022] A more dynamic signature deriving process is discussed with respect to Fig. L 
With reference to Fig. 1, an input signal is segmented in step 20. The signal may be an 
audio, video, or image signal, and may be divided into sets such as segments, frames, or 
blocks, respectively. Optionally, the sets may be further reduced into respective sub- 
sets. In step 22, the segmented signal is transformed into a frequency domain (e.g., a 
Fourier transform domain), or time-frequency domain. Applicable transformation 
techniques and related frequency-based analysis are discussed in Assignee's 
09/661,900 Patent Application, referenced above. Of course other frequency 
transformation techniques may be used. 

[0023] A transformed set's relevant features (e.g., perceptual relevant features 
represented via edges; magnitude peaks, frequency characteristics, etc.) are identified 
per set in step 24. For example, a set's perceptual features, such as an object's edges in 
a frame or a transition of such edges between frames, are identified, analyzed or 
calculated. In the case of a video signal, perceptual edges may be identified, analyzed, 
and/or broken into a defining map (e.g., a representation of the edge, the edge location 
relevant to the segment's orientation, and/or the edge in relation to other perceptual 
edges.). In another example, frequency characteristics such as magnitude peaks having 
a predetermined magnitude, or a relatively significant magnitude, are used for such 
identifying markers. These identifying markers can be used to form the relevant 
signature. 

[0024] Edges can also be used to calculate an object's center of mass, and the center 
of mass may be used as identifying information (e.g., signature components) for an 
object. For example, after thresholding edges of an object (e.g., identifying the edges), 
a centering algorithm may be used to locate an object's center of mass. A distance 
(e.g., up, down, right, left, etc.) may be calculated from the center of mass to each edge, 
or to a subset of edges, and such dimensions may be used as a signature for the object 
or for the frame. As an alternative, the largest object (or set of objects) may be selected 
for such center of mass analysis. 
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footer) of the segment, or otherwise be associated with the segment. In this case, the 
signature is preferably recoverable as the file is transferred, stored, transformed, etc. In 
another embodiment, a segment signature is stored in a segment header (or footer). The 
segment header may also be mathematically modified (e.g., encrypted with a key, 
XORed with an ID, etc.) for additional security. The stored content signature can be 
modified by the content in that segment, or hash of content in that segment, so that it is 
not recoverable if some or all of content is modified, respectively. The mathematical 
modification helps to prevent tampering, and to allow recovery of the signature in order 
to make a signature comparison. Alternatively, the signatures may be stored in a 
database instead of, or in addition to, being stored with the content item. The database 
may be local, or may be remotely accessed through a network such as a LAN, WAN, 
wireless network or internet. When stored in a database, a signature may be linked or 
associated with additional data. Additional data may include identifying information 
for the content (e.g., author, title, label, serial numbers, etc.), security information (e.g., 
copy control), data specifying actions or behavior (e.g., providing a URL, licensing 
information or rights, etc.), context information, metadata, etc. 

[0029] To illustrate one example, software executing on a user device (e.g., a 
computer, PVR, MP3 player, radio, etc.) computes a content signature for a content 
item (or segments within the content item) that is received or reviewed. The software 
helps to facilitate communication of the content signature (or signatures) to a database, 
where it is used to identify the related content item. In response, the database returns 
related information, or performs an action related to the signature. Such an action may 
include linking to another computer (e.g., a web site that returns information to the user 
device), transferring security or licensing information, verifying content and access, etc. 

[0030] Fig. 2 is a flow diagram illustrating one possible method to identify a content 
item from a stream of signatures (e.g., a linked set of consecutive derived signatures for 
an audio signal). In step 32, Viterbi decoding (as discussed further below) is applied 
according to the information supplied in the stream of signatures to resolve the identify 
of the content item. The Viterbi decoding efficiently matches the stream to the 
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with respect to Fig. 3. For example, if five signatures for a content item, each 
corresponding to a respective manipulation (or a series of manipulations) of the content 
item, are determined and stored with respect to a subject content item, then the same 
five signatures are preferably determined in step 52. The list of signatures is matched 
to the corresponding signatures stored in the database. As an alternative embodiment, 
subsets or levels of signatures may be matched (e.g., only 2 of the five signatures are 
derived and then matched). The security and verification confidence increases as the 
number of signatures matched increases. 

[0033] A set of perceptual features of a segment (or a set of segments) can also be 
used to create "fragile" signatures. The number of perceptual features included in the 
signature can determine its robustness. If the number is large, a hash could be used as 
the signature. 

Digital Watermarks and Content Signatures 

[0034] Content signatures may be used advantageously in connection with digital 
watermarks. 

[0035] A digital watermark may be used in conjunction with a content signature. The 
watermark can provide additional information, such as distributor and receiver 
information for tracking the content. The watermark data may contain a content 
signature and can be compared to the content signature at a later time to determine if 
the content is authentic. A content signature also can be compared to digital watermark 
data, and if the content signature and digital watermark data match (or otherwise 
coincide) the content is determined to be authentic. If different, however, the content is 
considered modified. 

[0036] A digital watermark may be used to scale the content before deriving a content 
signature of the content. Content signatures are sensitive to scaling (and/or rotation, 
distortion, etc.). A watermark can include a calibration and/or synchronization signal 



WO 02/51063 



PCT/US01/50238 



-12- 

[0039] To further protect a signature (e.g., stored in a header or digital watermark), a 
content owner could define a list of keys, which are used to scramble (or otherwise 
encrypt) the signature. The set of keys may optionally be based upon a unique ID 
associated with the owner. In this embodiment, a signature detector preferably knows 
the key, or gains access to the key through a so-called trusted third party. Preferably, it 
is optimal to have a signature key based upon content owner ID. Such a keying system 
simplifies database look-up and organization. Consider an example centered on audio 
files. Various record labels may wish to keep the meaning of a content ED private. 
Accordingly, if a signature is keyed with an owner ID, the central database only needs 
to identify the record label's content owner ID (e.g., an ID for BMG) and then it can 
forward all BMG songs to a BMG database for their response. In this case, the central 
database does not need all of the BMG content to forward audio files (or ID's) to BMG, 
and does not need to know the meaning of the content ID. Instead, the signature 
representing the owner is used to filter the request. 

Content Signature Calculations 

[0040] For images or video, a content signature can be based on a center of mass of 
an object or frame, as discussed above. An alterative method is to calculate an object's 
(or frame's) center of mass is to multiply each pixel's .luminescence with its location 
from the lower left comer (or other predetermined position) of the frame, sum all pixels 
within the object or frame, and then divide by the average luminescence of the object or 
frame. The luminescence can be replaced by colors, and a center of mass can be 
calculated for every color, such as RGB or CMYK, or one color. The center of mass 
can be calculated after performing edge detection, such as high pass filtering. The 
frame can be made binary by comparing to a threshold, where a 1 represents a pixel 
greater than the threshold and a 0 represents a pixel less than the threshold. The 
threshold can be arbitrary or calculated from an average value of the frame color, 
luminescence, either before or after edge detection. The center of mass can produce a 
set of values by being calculated for segments of the frame, in images or video, or for 
frames over time in video. 
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one of the fingerprints stored in the database. Upon finding a match, play-out of a 
given advertisement is confirmed. 

[0046] Some fingerprinting technology may employ a "hash" function to yield the 
fingerprint. Others may take, e.g., the most significant bit of every 10 th sample value to 
generate a fingerprint. Etc., etc. A problem arises, however, if the content is distorted. 
In such case, the corresponding fingerprint may be distorted too, wrongly failing to 
indicate a match. 

[0047] In accordance with this aspect of the present invention, content is encoded 
with a steganographic reference signal by which such distortion can be identified and 
quantized. If the reference data in a radio broadcast indicates that the audio is 
temporally scaled (e.g., by tape stretch, or by psycho-acoustic broadcast compression 
technology), the amount of scaling can be determined. The resulting information can 
be used to compensate the audio before fingerprint analysis is performed. That is, the 
sensed distortion can be backed-out before the fingerprint is computed. Or the 
fingerprint analysis process can take the known temporal scaling into account when 
deriving the corresponding fingerprint. Likewise with distorted image and video. By 
such approaches, fingerprint technology is made a more useful technique. 

[0048] (Pending U.S. Patent Application No. 09/452,023, filed 1 1/30/99, details such 
a reference signal (sometimes termed a "grid" signal, and its use in identifying and 
quantizing distortion. Pending U.S. Patent Application No. 09/689,250 details various 
fingerprint techniques.). 

[0049] In a variant system, a watermark payload - in addition to the steganographic 
reference signal - is encoded with the content. Thus, the hash (or other fingerprint) 
provides one identifier associated with the content, and the watermark provides 
another. Either can be used, e.g., to index related information (such as connected 
content). Or they can be used jointly, with the watermark payload effectively 
extending the ID conveyed by the hash (or vice versa). 
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payload, may signal to a compliant device that it should undertake a fingerprint analysis 
of the object. 

[0056] In other arrangements, the fingerprint detection is performed routinely, rather 
than triggered by a watermark. In such case, the watermark can specify an action that a 
compliant device should perform using the fingerprint data. (In cases where a 
watermark triggers extraction of the fingerprint, a further portion of the wateimark can 
specify a further action.) For example, if the watermark bit has a "0" value, the device 
may respond by sending the fingerprint to a remote database; if the watermark bit has a 
"1" value, the fingerprint is stored locally. 

[0057] Still further, frail (or so-called fragile) watermarks can be used in conjunction 
with fingerprint technology. A frail or fragile watermark is designed to be destroyed, 
or to degrade predictably, upon some form of signal processing. In the current 
fingerprinting environment, if a frail watermark is detected, then a fingerprint analysis 
is performed; else not. And/or, the results of a fingerprint analysis can be utilized in 
accordance with information conveyed by a frail watermark. 

Content Signatures from Compressed Data 

[0058] Content signatures can be readily employed with compressed or 
uncompressed data content. One inventive method determines the first n significant 
bits (where n is an integer, e.g., 64) of a compression signal and uses the n bits as (or to 
derive) a signature for that signal. This signature technique is particularly 
advantageous since, generally, image compression schemes code data by coding the 
most perceptually relevant features first, and then coding relevantly less significant 
features from there. Consider JPEG 2000 as an example. As will be appreciated by 
those skilled in that art, JPEG 2000 uses a wavelet type compression, where the image 
is hierarchically sub-divided into sub-bands, from low frequency perceptually relevant 
features, to higher frequency lesser perceptually relevant features. Using the low 
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pattern in the frequency components. As such, when the cue signal is found, the 
content signature is partially calculated, especially if the content signature is calculated 
with content before the cue (which should be" saved in memory while searching for the 
cue signal). Other cue signals may include, e.g., I-frames, synchronization signals, and 
digital watermarks. 

[0063] In the broadcast monitoring application, where the presence and amount of 
content is measured, such as an advertisement on TV, timing accuracy (e.g., with a 1 
sec.) is required. However, cue signals do not typically occur on such a regular interval 
(e.g., 1 sec). As such, content signatures related to a cue signal can be used to identify 
the content, but the computation of the content to locate the cue signal elements are 
saved to determine timing within the identified content. For example, the cue signal 
may include the contrast of the center of the frame, and the contrast from frame to 
frame represents the timing of the waveform and is saved. The video is identified from 
several contrast blocks, after a specific cue, such as fade to black in the center. The 
timing is verified by comparing the pre-existing and future contrasts of the center frame 
to those stored in the database for the TV advertisement. 

[0064] Content signatures are synchronized between extraction for entry into the 
database and for extraction for identifying the unknown content by using peaks of the 
waveform envelope. Even when there is an error calculating the envelope peak, if the 
same error occurs at both times of extraction, the content signatures match since they 
are both different by the same amount; thus, the correct content is identified. 

List Decoding and Trellis Coded Quantization 

[0065] The following discussion details another method, which uses Trellis Coded 
Quantization (TCQ), to derive a content signature from a content item. Whereas the 
following discussion uses an image for an example, it will be appreciated by one of 
ordinary skill in the art that the concepts detailed below can be readily applied to other 
content items, such as audio, video, etc. For this example, an image is segmented into 
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[0069] In step 62 (Fig. 6), the initial signature (e.g., hash) is stored in a database. 
Preferably, the signature is associated with a content ID, which is associated with a 
desired behavior, information, or action. In this manner, a signature may be used to 
index or locate additional information or desired behavior. 

Recalculating Signatures for Matching in the Database 

.[0070] In a general scenario, a content signature (e.g., hash) is recalculated from the 
content item as discussed above with respect to Trellis Coded Quantization. 

[0071] In many cases, however, a content signal will acquire noise or other distortion 
as it is transferred, manipulated, stored, etc. To recalculate the distorted content 
signal's signature (e.g., calculate a signature to be used as a comparison with a 
previously calculated signature), the following steps may be taken. Generally, list 
decoding is utilized as a method to identify the correct signature (e.g., the undistorted 
signature). As will be appreciated by one of ordinary skill in the art, list decoding is a 
generalized form of Viterbi decoding, and in this application is used to find the most 
likely signatures for a distorted content item. List decoding generates X the most likely 
signatures for the content item, where X is an integer. To do so, a list decoding method 
finds the X shortest paths (e.g., signatures) through a related trellis diagram. The 
resulting X shortest paths are then used as potential signature candidates to find the 
original signature. 

[0072] As an alternative embodiment, and before originally computing the signature 
(e.g., for storage in the database), a calibration watermark is embedded in the content 
item, and possibly with one or more bits of auxiliary data. A signature is then 
calculated which represents the content with the watermark signal. The calibration 
watermark assists in re-aligning the content after possible distortion when recomputing 
a signature from a distorted signal. The auxiliary data can also be used as an initial 
index into the database to reduce the complexity of the search for a matching a 
signature. Database lookup time is reduced with the use of auxiliary data. 
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addition, the database can be searched in conventional methods, such as binary tree 
methods. 

[0078] Given that the fingerprint is of fixed size, it represents a fixed number space. 
For example, a 32-bit fingerprint has 4 billion potential values. In addition, the data 
entered in the database can be formatted to be a fixed size. Thus, any database entry 
can be found by multiplying the fingerprint by the size of the database entry size, thus 
speeding access to the database. 

Content Addressable Memory 

[0079] Another inventive alternative uses a database based on content addressable 
memory (CAM) as opposed to RAM. CAM devices can be used in network equipment, 
particularly routers and switches, computer systems and other devices that require 
content searching. 

[0080] Operation of a CAM device is unlike that of a RAM device. For RAM, a 
controller provides an address, and the address is used to access a particular memory 
location within the RAM memory array. The content stored in the addressed memory 
location is then retrieved from the memory array. A CAM device, on the other hand, is 
interrogated by desired content. Indeed, in a CAM device, key data corresponding to 
the desired content is generated and used to search the memory locations of the entire 
CAM memory array. When the content stored in the CAM memory array does not 
match the key data, the CAM device returns a "no match" indication. When the content 
stored in the CAM memory array matches the key data, the CAM device outputs 
information associated with the content. Further reference to CAM technology can be 
made to U.S. Patent Nos. 5,926,620 and 6,240,003. 

[0081] CAM is also capable of performing parallel comparisons between input 
content of a known size and a content table completely stored in memory, and when it 
finds a match it provides the desired associated output. CAM is currently used, e.g., for 
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identified. If not, the next sub-fingerprint is used in the above process since the first 
sub-fingerprint must have had an error. Once the correct file is identified, the correct 
action or web link can quickly be found with CAM or traditional RAM-based 
databases, using the unique content identification, possibly a number or content name. 

Varying Content 

[0087] Some content items may be represented as a sequence of N bit signatures, 
such as time varying audio and video content. A respective N bit signature may 
correspond to a particular audio segment, or video frame, such as an I frame. A 
database may be structured to accommodate such a structure or sequence. 

[0088] In one embodiment, a calibration signal or some other frame of reference (e.g., 
timing, I frames, watermark counter, auxiliary data, header information, etc.) may be 
used to synchronize the start of the sequence and reduce the complexity of the database. 
For example, an audio signal may be divided into segments, and a signature (or a 
plurality of signatures) may be produced for such segments. The corresponding 
signatures in the database may be stored or aligned according to time segments, or may 
be stored as a linked list of signatures. 

[0089] As an alternative, a convolution operation is used to match an un- 
synchronized sequence of hashes with the sequences of hashes in the database, such as 
when a synchronization signal is not available or does not work completely. In 
particular, database efficiency may be improved by a convolution operation such as a 
Fast Fourier Transform (FFT), where the convolution essentially becomes a 
multiplication operation. For example, a 1-bit hash may be taken for each segment in a 
sequence. Then to correlate the signatures, an inverse FFT is taken of the 1 -bit hashes. 
The magnitude peaks associated with the signatures (and transform) are analyzed. 
Stored signatures are then searched for potential matches. The field is further narrowed 
by taking progressively larger signatures (e.g., 4-bit hashes, 8-bit hashes, etc.). 
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course, subject matter disclosed under one section heading can be readily combined 
with subject matter under other headings. 

[0094] The methods, processes, and systems described above may be implemented in 
hardware, software or a combination of hardware and software. For example, the 
transformation and signature deriving processes may be implemented in a 
programmable computer running executable software or a special purpose digital 
circuit. Similarly, the signature deriving and matching process and/or database 
functionality may be implemented in software, electronic circuits, firmware, hardware, 
or combinations of software, firmware and hardware. The methods and processes 
described above may be implemented in programs executed from a system's memory (a 
computer readable medium, such as an electronic, optical, magnetic-optical, or 
magnetic storage device). 

[0095] The particular combinations of elements and features in the above-detailed 
embodiments are exemplary only. In view of the wide variety of embodiments to 
which the principles and features discussed above can be applied, it should be apparent 
that the detailed embodiments are illustrative only and should not be taken as limiting 
the scope of the invention. Rather, we claim as our invention all such modifications as 
may come within the scope and spirit of the following claims and equivalents thereof. 
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grouping the features per set to comprise a respective signature for each of the 
sets; and 

linking the respective signatures. 

8. A method of resolving a stream of content signatures, the content signatures 
corresponding to sets of a content item, said method comprising the steps of: 

applying Viterbi decoding according to the stream of content signatures; 
identifying a content item corresponding to the stream; and 
accessing information related to the content item. 

9. A method of generating a content signature from compressed data, the 
compressed data having m bits, said method comprising the steps of: 

extracting n of the most significant of the m bits, where m > n, and n and m are 
integers; and 

storing the n bits as the content signature. 

10. A method of generating a content signature from a content item comprising 
the steps: 

in a compressed domain, identifying scaling features of the data; and 
grouping the scaling features to form a content signature. 

11. A method of generating a content signature for a signal comprising the 
steps of: 

dividing the signal into at least one set; and 

identifying perceptual edges of the set, the edges comprising the signature of the 

set. 

12. A method of generating a content signature for a signal comprising the 
steps 

of: 
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19. The method of claim 18, wherein the content signature is used to identify 
the content item. 

20. The method of claim 19, further comprising the steps of selecting a 
database for interrogation based on the distributor identifier, and identifying 
information associated with the content item and stored in the selected database with 
the content signature. 

21. The method of claim 17, wherein the message comprises a content 
signature. 

22. The method of claim 21, further comprising the step of comparing the 
message content signature with derived content signature. 

23. The method of claim 22, further comprising the step of deeming the content 
item authentic when the message content signature and the derived content signature 
coincide. 

24. The method of claim 17, wherein the message comprises a trigger to 
indicate that said deriving step should be performed. 

25. A method to derive a content signature for a video frame or image 
comprising the steps of: 

identifying an area in the video frame or image; 
determining a center of mass of the video frame or image; and 
providing a content signature for the video frame or image based at least on the 
center of mass. 

26. The method of claim 25, wherein in the center of mass is determined by 
identifying edges of the area and then determining a center based on the identified 
edges. 
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36. A method of calculating a content signature from a content item, the 
content item comprising at least one cue signal, said method comprising the steps of: 

sensing the cue signal from the content item; and upon sensing the cue signal, 
determining a content signature for at least a portion of the content item. 

37. The method of claim 36, wherein the content item is video and the cue 
signal 

comprises a fade to black indicator. 

38. The method of claim 36, wherein the cue signal comprises a pattern of 
frequency components for the content item. 

39. The method of claim 36, wherein the content item is video and the cue 
signal 

comprises a contrast of a center of a video frame. 

40. The method of claim 36, further comprising the step of determining timing 
intervals within the content item based on the cue signal. 

41 . A data management method comprising the step of: 
deriving a content signature from a content item; and 

providing the content signature to a database constructed as content addressable 
memory (CAM); and 

obtaining data from the database associated with the content signature. 

42. The method of claim 42, wherein the data comprises at least one of a URL, 
IP address and metadata. 
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